SherbyNode - /page/gopher.mu

Micron Document

Gopher Proxy

Main menu Proxy Menu Search Engine Back

gopherpedia.com:70 gopherpedia.com:70/0//DeepSeek

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

======================================================================

= DeepSeek =

======================================================================

Introduction

======================================================================

Hangzhou DeepSeek Artificial Intelligence Basic Technology Research

Co., Ltd., doing business as DeepSeek, is a Chinese artificial

intelligence (AI) company that develops open weights large language

models (LLMs). Based in Hangzhou, Zhejiang, DeepSeek is owned and

funded by High-Flyer, a Chinese hedge fund. DeepSeek was founded in

July 2023 by Liang Wenfeng, who serves as the CEO for both of the

companies. The company launched an eponymous chatbot alongside its

DeepSeek-R1 model in January 2025.

DeepSeek-R1 provided responses comparable to other contemporary LLMs,

such as OpenAI's GPT-4 and o1. Its training cost was reported to be

lower than other LLMs. The company claims that it trained V3 for US$6

million, which was less than OpenAI's reported US$100 million cost for

GPT-4 in 2023. DeepSeek also claimed that they were using

approximately one-tenth the computing power consumed by Meta's

comparable model, Llama 3.1. DeepSeek's success against larger and

more established rivals has been described as "upending AI".

DeepSeek's models are open-weight, meaning that the exact parameters

are openly shared, but the training data is not openly licensed. Since

the January 2025 debut of DeepSeek-R1, the company has made its new

models available under free and open-source software licenses, like

the MIT License. The company reportedly recruits AI researchers from

top Chinese universities and hires from outside traditional computer

science fields to broaden its models' knowledge and capabilities.

DeepSeek is considered more closely linked to the defense industry of

China, due to previous affiliations of its researchers, and its

chatbot's non-combat role adoption by the People's Liberation Army

since March 2025.

R1 was geopolitically significant as a open-weight, cost-effective,

and high-performing release. The company also trained its models

during ongoing US semiconductor trade restrictions on China, using a

fewer number of weaker export model chips via a mixture of experts

approach. Observers like the New York Post and The Guardian described

R1 as a Sputnik moment in threatening the proprietary-led US AI

ecosystem, including Nvidia, which lost US$600 billion in market

value, the largest single-company decline in US stock market history.

Pre-founding years

====================

In June 2015, High-Flyer was co-founded by AI enthusiast Liang

Wenfeng, who had been trading since the 2008 financial crisis while

attending Zhejiang University. The company began stock trading using a

GPU-dependent deep learning model on 21 October 2016. Previously, it

had used CPU-based linear models. By the end of 2017, most of its

trading was driven by AI.

Fire-flyer

============

In 2019, the company began constructing its first computing cluster,

Fire-Flyer, at a cost of 200 million yuan. The computing cluster

contained 1,100 GPUs interconnected at 200 Gbit/s and was retired

after 1.5 years in operation. By 2021, Liang had started buying large

quantities of Nvidia GPUs for an AI project, reportedly obtaining

10,000 Nvidia A100 GPUs before the United States restricted chip sales

to China.

Fire-flyer 2

==============

Computing cluster Fire-Flyer 2 began construction in 2021 with a

budget of 1 billion yuan. It was reported that in 2022, Fire-Flyer 2's

capacity had been used at over 96%, totaling 56.74 million GPU hours.

27% of Fire-Flyer 2's capacity was used to support scientific

computing outside the company. Fire-Flyer 2 had 5,000 PCIe A100 GPUs

in 625 nodes, each containing 8 GPUs. At the time, it exclusively used

PCIe instead of the DGX version of A100. This was because the models

it trained could fit within a single 40 GB GPU VRAM. Hence, there was

no need for the higher bandwidth of DGX at the time (it required only

data parallelism but not model parallelism). Later, it incorporated

both NVLinks and Nvidia Collective Communications Library (NCCL) to

train larger models that required model parallelism.

Founding

==========

On 14 April 2023, High-Flyer announced the launch of an artificial

general intelligence (AGI) research lab, stating that the new lab

would focus on developing AI tools unrelated to the firm's financial

business. Two months later, on 17 July 2023, that lab was spun off

into an independent company. That lab was named DeepSeek, with

High-Flyer as its principal investor and backer. Initially, venture

capital investors were reluctant to provide funding, as they

considered it unlikely that the venture would be able to quickly

generate an exit (an ownership stake in an investment).

Strategy

==========

DeepSeek has stated that it focuses on research and does not have

immediate plans for commercialization. This posture also means it can

skirt certain provisions of China's AI regulations aimed at

consumer-facing technologies.

DeepSeek's hiring approach emphasizes skills over lengthy work

experience, resulting in many hires fresh out of university. The

company likewise recruits individuals without computer science

backgrounds to expand the range of expertise incorporated into the

models, for instance in poetry or advanced mathematics. According to

The New York Times, dozens of DeepSeek researchers have or have

previously had affiliations with People's Liberation Army laboratories

and the Seven Sons of National Defence. Since 2025, its chatbot was

adopted in non-combat roles by the People's Liberation Army, including

hospitals, the People's Armed Police paramilitary, and national

mobilization organizations.

Due to U.S. chip restrictions, DeepSeek has continuously refined its

algorithms to maximize computational efficiency, leveraging older

hardware and reducing energy consumption.

DeepSeek also expanded on the African continent as it offers more

affordable and less power-hungry AI solutions. The company has

bolstered African language models and generated a number of startups,

for example in Nairobi. Along with Huawei's storage and cloud

computing services, the impact on the tech scene in sub-Saharan Africa

is considerable. DeepSeek offers local data sovereignty and more

flexibility compared to Western AI platforms.

Training framework

======================================================================

High-Flyer/DeepSeek had operated at least two primary computing

clusters: Fire-Flyer and Fire-Flyer 2. Fire-Flyer 1 was constructed in

2019 and was retired after 1.5 years of operation. Fire-Flyer 2 is

still in operation as of 2025. Fire-Flyer 2 consists of co-designed

software and hardware architecture. On the hardware side, Nvidia GPUs

use 200 Gbps interconnects. The cluster is divided into two zones, and

the platform supports cross-zone tasks. The network topology was two

fat trees, chosen for high bisection bandwidth. On the software side,

there are:

*3FS (Fire-Flyer File System): A distributed parallel file system

designed for asynchronous random reads. It uses Direct I/O and RDMA

Read. In contrast to standard Buffered I/O, Direct I/O does not cache

data. Caching is useless in this case, since each piece of data read

is random and is not reused.

*hfreduce: Library for asynchronous communication, originally designed

to replace Nvidia Collective Communication Library (NCCL). It is

mainly used for allreduce, especially of gradients during

backpropagation. It runs asynchronously on the CPU to avoid blocking

kernels on the GPU. It uses two-tree broadcast like NCCL.

*hfai.nn: Software library of commonly used operators for neural

network training, similar to torch.nn in PyTorch.

*HaiScale Distributed Data Parallel (DDP): Parallel training library

that implements various forms of parallelism such as data parallelism

(DP), pipeline parallelism (PP), tensor parallelism (TP), experts

parallelism (EP), fully sharded data parallel (FSDP) and zero

redundancy optimizer (ZeRO). It is similar to PyTorch DDP, which uses

NCCL on the backend.

*HAI Platform: Various applications such as task scheduling, fault

handling, and disaster recovery.

As of 2022, Fire-Flyer 2 had 5,000 PCIe A100 GPUs in 625 nodes, each

containing 8 GPUs. It later incorporated NVLinks and NCCL to train

larger models that required model parallelism.

Model releases

======================================================================

DeepSeek Coder and DeepSeek LLM

=================================

DeepSeek released its first model, DeepSeek Coder, on 2 November 2023.

The DeepSeek-Coder V2 series was released shortly after. It was

followed by the DeepSeek-LLM series on 29 November 2023.

Misc

======

In January 2024, DeepSeek released two DeepSeek-MoE models. On 3 April

2024, they released 3 DeepSeek-Math models.

V2 series

===========

DeepSeek-V2 was released in May 2024. In September 2024, DeepSeek-V2.5

was introduced and revised in December.

V3 series

===========

In December 2025, DeepSeek-V3-Base and DeepSeek-V3 (chat) were

released. On 24 March 2025, DeepSeek released DeepSeek-V3-0324 under

the MIT License, a revised version of V3.

On 21 August 2025, DeepSeek released DeepSeek V3.1 under the MIT

License. This model features a hybrid architecture with thinking and

non-thinking modes. It also surpasses prior models like V3 and R1, by

over 40% on certain benchmarks like SWE-bench and Terminal-bench. It

was updated to V3.1-Terminus on 22 September 2025, which fixed

multiple language artifacts and improved in intelligence in general.

DeepSeek V3.2-Exp was released on 29 September 2025. It uses DeepSeek

Sparse Attention, a more efficient attention mechanism based on

previous research published in February. It was based on DeepSeek V3.1

Terminus. DeepSeek-V3.2 was released on 1 December 2025, alongside a

DeepSeek-V3.2-Speciale variant that focused on reasoning.

R1 series

===========

On 20 November 2024, the preview of DeepSeek-R1-Lite became available

via chat. It was a proprietary model, having only ever been released

via API and web. On 20 January 2025, DeepSeek launched the DeepSeek-R1

model, which was free for iOS and Android. By 27 January, DeepSeek

surpassed ChatGPT as the most downloaded freeware app on the iOS App

Store in the United States, triggering an 18% drop in Nvidia's share

price. DeepSeek subsequently released multiple distilled variants of

DeepSeek R1, ranging from 1.5B parameters to 70B parameters.

On 28 May 2025, DeepSeek released DeepSeek-R1-0528 under the MIT

License, which was a minor update upon R1. Its scores have also

exceeded both DeepSeek R1 (January) and DeepSeek V3-0324 in certain

benchmarks.

V4 series

===========

On 24 April 2026, DeepSeek released a preview of its V4 series,

including the 284-billion parameter DeepSeek-V4-Flash and the

1.6-trillion parameter DeepSeek-V4-Pro. Both feature a one million

token context window, under the MIT License. V4 has been adopted by

semiconductor manufacturers such as Huawei and Cambricon Technologies.

The official versions of DeepSeek V4-Flash and V4-Pro were released on

31 July and 13 August, respectively.

List of models

================

Major versions of DeepSeek models. SFT stands for supervised

finetuning, which allows AI labs to precisely adjust an AI model's

behavior.

!Major versions !Release date !Status !Major variants !License

!Remarks

|DeepSeek-Coder | | |Base (pretrained) Instruct (with

instruction-finetuned) |rowspan=10 |The architecture is essentially

the same as Llama.

essentially the same as Llama.

rowspan="3" |DeepSeek-Math rowspan="3" | rowspan="3" |Base

|Initialized with DS-Coder-Base-v1.5

|Instruct (with SFT) |

|RL (using a process reward model) |Developed Group Relative Policy

Optimization (GRPO), a variant of Proximal Policy Optimization (PPO).

|DeepSeek-V2 | | |DeepSeek-V2, DeepSeek-V2-Chat DeepSeek-V2-Lite,

DeepSeek-V2-Lite-Chat DeepSeek-Coder-V2 DeepSeek-V2.5 |Developed

multi-head latent attention (MLA). Also used MoE. Implemented KV

caching.

|DeepSeek-V3 | | |DeepSeek-V3-Base DeepSeek-V3 (a chat model) |The

architecture is essentially the same as V2. Updated on 2025-03-24.

|DeepSeek-Prover-V2 | | |DeepSeek-Prover-V2-671B

DeepSeek-Prover-V2-7B |

|DeepSeek-VL2 | | |

version, only accessed through API and a chat interface.

rowspan="2" | rowspan="2" |DeepSeek-R1 DeepSeek-R1-Zero

DeepSeek-R1-0528 rowspan=5 |Initialized from DeepSeek-V3-Base and

sharing the V3 architecture.

|Distilled models |Initialized from other models, such as Llama,

Qwen, etc. Distilled from data synthesized by R1 and R1-Zero.

| | |DeepSeek-R1-0528 |N/A

rowspan="2" |DeepSeek-V3.1 | | |DeepSeek-V3.1-Base DeepSeek-V3.1 (a

chat model) |Hybrid architecture (thinking and non-thinking modes

available). Trained on over 800B additional tokens on top of V3.

| | |DeepSeek-V3.1-Terminus |Reducing instances of mixed

Chinese-English text and occasional abnormal characters on top of

V3.1.

|DeepSeek-Math-V2 | | | |

|DeepSeek-V3.2 | | |DeepSeek-V3.2 DeepSeek-V3.2-Speciale rowspan=2 |

Official release of V4-Flash on 31 July, and V4-Pro on 13 August

The first DeepSeek models were essentially the same as Llama, which

were dense decoder-only transformers. Later models incorporated the

multi-head latent attention (MLA), MoE, and KV caching.

A decoder-only transformer consists of multiple identical decoder

layers. Each of these layers feature two main components: an attention

layer and a feedforward network (FFN) layer. V2 replaced the standard

multi-head attention mechanism (MHA) with multi-head latent attention

(MLA). This introduces compressed latent vectors to reduce key value

(KV) cache size, and thus memory usage.

A standard Mixture of Experts Transformer generally use the

sparsely-gated MoE layers in the FFN layers. In such an MoE layer,

there are several FFN modules in parallel (routed experts) and a small

classifier (gate) to compute a score for all these modules upon each

token. Only the highest-scoring modules are activated. Starting with

DeepSeekMoE, DeepSeek adopted a variant that adds shared experts,

which are always activated.

Technical specifications of models

======================================================================

DeepSeek's models are open-weight, which provides less freedom for

modification than true open source software.

DeepSeek Coder

================

DeepSeek Coder is a series of eight models, four pretrained (Base) and

four instruction-finetuned (Instruct). All have 16K context lengths.

The model was made open-weights under the DeepSeek License, which

includes restrictions like open and responsible downstream usage.

The training program was:

# Pretraining: 1.8T tokens (87% source code, 10% code-related English

(GitHub markdown and Stack Exchange), and 3% code-unrelated Chinese).

# Long-context pretraining: 200B tokens. This extends the context

length from 4K to 16K. This produced the Base models.

# Supervised finetuning (SFT): 2B tokens of instruction data. This

produced the Instruct models.

They were trained on clusters of A100 and H800 Nvidia GPUs, connected

by InfiniBand, NVLink, NVSwitch.

DeepSeek Coder properties

!! Layers !! Model !Intermediate !! # Heads !! # Kv-heads

1.3B 24 2048 |5504 16 16

|5.7B |32 |4096 |11008 |32 |1

6.7B 32 4096 |11008 32 32

33B 62 7168 |19200 56 7

DeepSeek-LLM

==============

The DeepSeek-LLM series was released in November 2023. It has 7B and

67B parameters in both Base and Chat forms. DeepSeek's accompanying

paper claimed benchmark results higher than Llama 2 and most

open-source LLMs at the time. The model code is under the

source-available DeepSeek License.

The architecture was essentially the same as the Llama series. Both

had vocabulary size 102,400 (byte-level BPE) and context length of

4096. They trained on 2 trillion tokens of English and Chinese text

obtained by deduplicating the Common Crawl.

DeepSeek LLM properties

.. !! # Layers !! Model !Intermediate !! # Heads !! # Kv-heads

7B 30 4096 |11008 32 32

67B 95 8192 |22016 64 8

The chat versions of the two base models were released concurrently,

obtained by training base by supervised finetuning (SFT) followed by

direct policy optimization (DPO). DPO is also known as reinforcement

learning from human feedback.

DeepSeek-MoE

==============

DeepSeek-MoE models (Base and Chat), each have 16B parameters (2.7B

activated per token, 4K context length). The training was essentially

the same as DeepSeek-LLM 7B, and was trained on a part of its training

dataset. DeepSeek claimed performance comparable to a 16B MoE as a 7B

non-MoE. It is a variant of the standard sparsely-gated MoE, with

shared experts that are always queried, and routed experts that might

not be. The company found this to help with expert balancing. In

standard MoE, some experts can become overused, while others are

rarely used, which wastes space. Attempting to balance expert usage

causes experts to replicate the same capacity. They proposed that the

shared experts to learn core capacities that are often used, and let

the routed experts learn peripheral capacities that are rarely used.

DeepSeek-Math

===============

DeepSeek-Math includes 3 models: Base, Instruct, and RL. Math was

trained as follows:

# Initialize with a previously pretrained DeepSeek-Coder Base v1.5 7B.

# Further pretrain with 500B tokens (6% DeepSeekMath Corpus, 4%

AlgebraicStack, 10% arXiv, 20% GitHub code, 10% Common Crawl). This

produced Base.

# Train an instruction-following model by SFT Base with 776K math

problems and tool-use-integrated step-by-step solutions. This produced

Instruct.

# Reinforcement learning (RL) The reward model was a process reward

model (PRM) trained from Base according to the Math-Shepherd method.

This reward model was then used to train Instruct using group relative

policy optimization (GRPO) on a dataset of 144K math questions related

to GSM8K and MATH. The reward model was continuously updated during

training to avoid reward hacking. This resulted in RL.

====

In May 2024, DeepSeek released the DeepSeek-V2 series. The series

includes 4 models, 2 base models (DeepSeek-V2, DeepSeek-V2 Lite) and 2

chatbots (chat variants). The two larger models were trained as

follows:

# Pretrain on a dataset of 8.1T tokens, using 12% more Chinese tokens

than English ones.

# Extend context length from 4K to 128K using YaRN. This resulted in

DeepSeek-V2.

# SFT with 1.2M instances for helpfulness and 0.3M for safety. This

resulted in Chat SFT, which was not released.

# RL using GRPO in two stages. The first stage was trained to solve

math and coding problems. This stage used 1 reward model, trained on

compiler feedback (for coding) and ground-truth labels (for math). The

second stage was trained to be helpful, safe, and follow rules. This

stage used 3 reward models. The helpfulness and safety reward models

were trained on human preference data. The rule-based reward model was

manually programmed. All trained reward models were initialized from

Chat (SFT). This resulted in the released version of Chat.

They opted for 2-staged RL, because they found that RL on reasoning

data had unique characteristics different from RL on general data. For

example, RL on reasoning could improve over more training steps.

The two V2-Lite models were smaller, and trained similarly.

DeepSeek-V2 Lite-Chat underwent only SFT, not RL. They trained the

Lite version to help further research and development on MLA and

DeepSeekMoE.

Architecturally, the V2 models were significantly different from the

DeepSeek LLM series. They changed the standard attention mechanism by

a low-rank approximation called multi-head latent attention (MLA), and

used the previously published MoE variant.

DeepSeek V2 properties

!Name . !Active !! # Layers!! Context length !# Shared experts!! #

Routed experts

|V2-Lite 15.7B |2.4B 27 32K |2 64

|V2 236B |21B 60 128K |2 160

The 'Financial Times' reported that it was cheaper than its peers with

a price of 2 RMB for every million output tokens.

The DeepSeek-Coder V2 series included V2-Base, V2-Lite-Base,

V2-Instruct, and V2-Lite-Instruct. Training process:

# Base models were initialized from corresponding intermediate

checkpoints after pretraining on 4.2T tokens (not the version at the

end of pretraining), then pretrained further for 6T tokens, then

context-extended to 128K context length.

# DeepSeek-Coder and DeepSeek-Math were used to generate 20K

code-related and 30K math-related instruction data, then combined with

an instruction dataset of 300M tokens. This was used for SFT.

# RL with GRPO. The reward for math problems was computed by comparing

with the ground-truth label. The reward for code problems was

generated by a reward model trained to predict whether a program would

pass the unit tests.

DeepSeek-V2.5 was made by combining DeepSeek-V2-Chat and

DeepSeek-Coder-V2-Instruct.

====

DeepSeek-V3-Base and DeepSeek-V3 (chat variant) essentially uses the

same architecture as V2 with the addition of multi-token prediction,

which optionally decodes extra tokens faster but less accurately.

Training process:

# Pretraining on 14.8T tokens of a multilingual corpus, mostly English

and Chinese. It contained a higher ratio of math and programming than

the pretraining dataset of V2.

# Extend context length twice, from 4K to 32K and then to 128K, using

YaRN. This produced DeepSeek-V3-Base.

# SFT for 2 epochs on 1.5M samples of reasoning (math, programming,

logic) and non-reasoning (creative writing, roleplay, simple question

answering) data. Reasoning data was generated by expert models.

Non-reasoning data was generated by DeepSeek-V2.5 and checked by

humans.

#* The expert models were trained by starting with an unspecified base

model, then SFT on both data, and synthetic data generated by an

internal DeepSeek-R1-Lite model. The system prompt asked R1 to reflect

and verify during thinking. Then the expert models were RL using an

undisclosed reward function.

#* Each expert model was trained to generate just synthetic reasoning

data in one specific domain (math, programming, logic).

#* Expert models were used instead of R1 itself, since the output from

R1 itself suffered overthinking, poor formatting, and excessive

length.

# Model-based reward models were made by starting with a SFT

checkpoint of V3, then finetuning on human preference data containing

both final reward and chain-of-thought leading to the final reward.

The reward model produced reward signals for both questions with

objective but free-form answers, and questions without objective

answers (such as creative writing).

# An SFT checkpoint of V3 was trained by GRPO using both reward models

and rule-based reward. The rule-based reward was computed for math

problems with a final answer (put in a box), and for programming

problems by unit tests. This produced DeepSeek-V3.

DeepSeek released its DeepSeek-V3-0324 model, which used the same

architecture as V3, on 24 March 2025 under the MIT License.

DeepSeek V3 properties

!Name . !Active !! # Layers !! Context length !# Shared experts!! #

Routed experts

|V3 671B |37B 61 128K |1 256

The DeepSeek team performed low-level engineering to improve

efficiency, such as mixed-precision arithmetic. Much of the forward

pass was performed in 8-bit floating point numbers (5E2M: 5-bit

exponent and 2-bit mantissa) rather than the standard 32-bit,

requiring special general matrix multiply (GEMM) routines to

accumulate accurately. They used a custom 12-bit float (E5M6) only for

the inputs to the linear layers after the attention modules. Optimizer

states were in 16-bit (BF16). They minimized communication latency by

extensively overlapping computation and communication, such as

dedicating 20 streaming multiprocessors out of 132 per H800 for only

inter-GPU communication. They lowered communication by rearranging

(every 10 minutes) the exact machine each expert was on so as to avoid

querying certain machines more often than others, adding auxiliary

load-balancing losses to the training loss function, and other

load-balancing techniques.

After training, it was deployed on clusters of H800 GPUs. The 8 H800

GPUs within a cluster were connected by NVLink, and the clusters were

connected by InfiniBand.

Total cost of training the DeepSeek-V3 model

Stage Cost (in one thousand GPU hours) Cost (in one million US$)

Pre-training 2,664 5.328

Context extension 119 0.24

Fine-tuning 5 0.01

Total 2,788 5.576

The cost has been discussed and called misleading, because it covers

only parts of the true cost.

Benchmark tests show that V3 outperformed Llama 3.1 and Qwen 2.5 while

matching GPT-4o and Claude 3.5 Sonnet.

====

In January 2025, DeepSeek released the DeepSeek-R1 model under the MIT

License.

DeepSeek-R1-Lite-Preview was trained for logical inference,

mathematical reasoning, and real-time problem-solving. DeepSeek

claimed that it exceeded performance of OpenAI o1 on benchmarks such

as American Invitational Mathematics Examination (AIME) and MATH.

However, 'The Wall Street Journal' reported that on 15 problems from

the 2024 edition of AIME, the o1 model reached a solution faster.

DeepSeek-R1 and DeepSeek-R1-Zero were initialized from

DeepSeek-V3-Base and share its architecture. DeepSeek-R1-Distill

models were instead initialized from other pretrained open-weight

models, including LLaMA and Qwen, then fine-tuned on synthetic data

generated by R1.

DeepSeek-R1-Zero was trained exclusively using GRPO RL without SFT.

Unlike previous versions, it used no model-based reward. All reward

functions were rule-based, mainly of two types (other types were not

specified): accuracy rewards and format rewards. Accuracy reward was

checking whether a boxed answer is correct (for math) or whether a

code passes tests (for programming). Format reward was checking

whether the model puts its thinking trace within a ... tag.

R1-Zero has issues with readability and mixing languages. R1 was

trained to address these issues and further improve reasoning:

# SFT DeepSeek-V3-Base on thousands of cold-start data all with the

standard format of |special_token||special_token|, designed to improve

model output readability.

# Apply the same GRPO RL process as R1-Zero, adding a language

consistency reward to encourage it to respond monolingually. This

produced an unreleased internal model.

# Synthesize 600K reasoning data from the internal model, with

rejection sampling (i.e. if the generated reasoning had a wrong final

answer, then it is removed). Synthesize 200K non-reasoning data

(writing, factual QA, self-cognition, translation) using DeepSeek-V3.

# SFT DeepSeek-V3-Base on the 800K synthetic data for 2 epochs.

# Apply the same GRPO RL process as R1-Zero with rule-based reward

(for reasoning tasks), but also model-based reward (for non-reasoning

tasks, helpfulness, and harmlessness). This produced DeepSeek-R1.

Distilled models were trained by SFT on 800K data synthesized from

DeepSeek-R1, in a similar way as step 3. They were not trained with

RL.

====

In April 2026, DeepSeek released a preview of their new V4 model

series, also under the MIT License.

The DeepSeek V4 model improved on their previous V3/R1 architecture in

the following ways:

*The model uses their Manifold-constrained Hyper Connections (mHC)

architecture, which is claimed by DeepSeek to enhance conventional

residual connections.

*They introduced Constrained Sparse Attention (CSA) and Heavily

Compressed Attention (HCA), modifications to the Attention mechanism

used within Transformer models based on their prior DeepSeek Sparse

Attention architecture introduced in V3.2.

*The Muon optimizer was used for most layers for faster convergence

and improved training stability.

They released two sizes of model, V4-Flash and V4-Pro. Each are able

to be operated in a non-reasoning mode, a reasoning mode, and a "Max"

extended reasoning mode.

DeepSeek V4 Models

Parameter !! DeepSeek-V4-Flash !! DeepSeek-V4-Pro

Total parameters 284B 1.6T

Active parameters 13B 49B

Layer count 43 61

Hidden dimension size 4096 7168

Routed MoE Experts 256 384

Legal status

======================================================================

In the United States, the National Defense Authorization Act for

Fiscal Year 2026 instructs the United States Secretary of Defense and

the Director of National Intelligence to remove and exclude any

artificial intelligence developed by DeepSeek or owned by HighFlyer

from devices operated by the United States Department of Defense and

United States Intelligence Community or their contractors. Artificial

intelligence by DeepSeek is only allowed with explicit permissions for

research, training, and evaluation or military activities supporting

national security functions such as counterterrorism or

counterintelligence.

In Australia, the Department of Home Affairs issued a government-wide

directive "to prevent the use or installation of DeepSeek products,

applications and web services and where found remove all existing

instances of DeepSeek products, applications and web services from all

Australian Government systems and devices."

See also

======================================================================

* List of AI-assisted software development tools

* List of LLMs

* Lists of open-source artificial intelligence software

External links

======================================================================

* [https://github.com/deepseek-ai DeepSeek] on GitHub

* [https://huggingface.co/deepseek-ai/DeepSeek-V2.5-1210 DeepSeek] on

Hugging Face

* [https://api-docs.deepseek.com/ Official API documentation]

[https://huggingface.co/collections/Presidentlin/deepseek-papers-674c536aa6acddd9bc98c2ac

Anthology of DeepSeek papers]

* [https://www.high-flyer.cn/blog/ Research blog of High-Flyer]

License

=========

All content on Gopherpedia comes from Wikipedia, and is licensed under CC-BY-SA

License URL: http://creativecommons.org/licenses/by-sa/3.0/

Original Article: http://en.wikipedia.org/wiki/DeepSeek

you're on gopherpedia.com^70/0//DeepSeek

back link is gopherpedia.com:70